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IDENTIPIEB5 , , Continuing Conference for the Liberal Arts ' 

AB.STFACT ' ' . . . - . , . ^ 

The Conti-nuing Conference • for the LibeBal Arts was 
organized to test the thesis th^t Interdisciplinary studies involving 
che b^sic arts and sciences can be structured in a way t6 provide 
superior -prep^ation fpr immediate entry into careers not always- ^ 
associated>.wj^ undergraduate .education,. The programs of Augustafaa 
College, D^nison University, _and Saint Joseph's .College are described 
as Illustrative of the yatiety of programs 'maintained by- Conference 
Institutions.^ This paper attempts^,to formulate a model tp evalua^te 
these diver^B programs taking into considetatio?i .the primacy of 
persotiality development and attitudinal changes within tbe programs 
u'Tlder study. A model Is proposed with these desired -OutcKMjes : (1) 
collection bf data on wh4.ch to ground decision- making with reasonable 
expenditure pf resources; (2) Tdocuiaentation of evidence so that ' 
outride eyaluators ca-n jiake guasi-independent judgments^ and be 
persuaded .by the statements of outsiders; an^, (3) cooperatioja, with 
program staff wit h- regard , to tha dev^elopment af evaluation design, 
s^lf-eva-luation, and report writing. (PC) ' ' . s . 
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Thgf Continuing .Conference^ for' the^ Liberal Arts sponsorec^ by.; ' 
the Lilly Endowment/ Inc.; was organized to test the thesis that 
inter-disciplinary studies irtvotving^ the basic arts and scilence^ 
•can be structured in a way to provide superior preparation, fgr 
immediate entry into* careers npt always' associated with undergr^uate 
education • * As a result gf a /series of meetings a new set of ^ " 
relationships has emerged among the colleges, relationships which 
have already fd'stered plans for exchange of ideas, of- people, of ^ 
programs an(3 of resources. ^Of particular concern to the twenty 
schools Iftrrm^ng the Conference is the attention given - to implement- 
. mg programs ror career develdpment within the context of value^ ' 
and a ^'traditional liberal arts curriculum, - ' ^ ' , ' ^ 

Illustrati<re of the variety oS.,^^^grams maintained by Cort-* . 
iinuing Conference institutions are the following from Ai3gusta(na*\^ 
^CoiJ-lege, Denlson University, and Saint Joseph's College. - ^ . . , ^ 

The Augiistana program comlijines shqrt periods of of f -campus - 
work-study with basic academic programs pjci campus for both students 
and faQulty, with the intent of increasing career aw^re?ness arfiong 
both. Students spend four/.week^.ph^the job with lo^fcal employers 
including an agricultural equipment rtianuf acturer , radio and TV 
stations and museums. The internships are supplemented vith a / 
^seminar and an evaluation* period. JB'aculty are placed in situations 
sp.milar to student internships.^ The intent is to 'enlarge ' the * 
faculty's knowledge of the business and professional wcji^ld.^Cand 
their value, systems) and to help them be more ef fecti/ve' in advising ' 
students regardi^cf career choices* ^ * . ^ / 




uite 'possible to develop a 
ig, and th^ quality of an 



^he Denison, University program on "Simulatior\ and Le'ar^iing" 
intends^!K44^§g^^^^ between theory^. and the application of what, 
is learned. It provides intensive ^raiAfng in problem-solving and 

^decision-rmaking* l£ creates contexts in-'wiith complex questions 

'"-^^ • ■ : • . X . . 

of va**!^ may be addTessed "^re effectively.. .' Student activities 

include role^=»^aying, sin\ulation^gaVnes and modeling ^.o^ complex 

situations. It 'apj 

working relat/lonship betwe? 

individual's life, 

At* S^int Joseph's the approach^ to'.yalUes atnd^ careers is 

• ' . * ' " • ^ - ^ ' ' *^ - 

abased on an interdisc'iplinaty program which • is in^e^grativ^ ih its - 

Structure. It gives* "^the ^nttrre. student/ body -a cQirnnoji experience in 

' ^ ' . ^ ' ^ \ ^ ! . : , .* ~ % ' , 

reflecting on man, his situ^tioitV his\ e/clii^veraQi)ts and .problems, 

' " J . ^ . * ' ^ I ■ ' : '/ . . ' . 

his meaning "and purpose. .The mdve to- Core demanded radic5al changes 

in schedules, in departmental offerings^ irV -dourse assignments/ 
» ' ^ • • . / * , <\ ' . 

and in many other 'ictUg held policies /and ideas. 

^ In attiempting to formulate a model to evaluate such diverse 

programs, we faced various obstacles, which prevented us fjciom addpt- 

' ' • ' - ' ' O * ' i' " ^ * • 

ing, several existanb proC%d\if es developed by^ educational researchers 
^The' primacy of personality .development ajid attitudinacl changes, 
within the programs under study precluded jasing a^ny single measure-, 
ments/of 'cog;iitiv,e achievement. We needed to assess behavioral - 
and ,attitudinal changes./ regardless, of whether this 'occurred during 
a career-oriented program, typified by driternsh'ips , or a cultural 
"and value-orignted curriculumi^'as ^exemplif ied by Saint Joseph's i 



College.- In many instances the es-t^iblished' g^ls set by the staff 
lay outside the, area of measurement, that is, they were abstract 
and global rather than specif i"s^and behavioral. In this the - 
educators- involved manifested a'commpn attitude, stemming certainly 
from the time of Clasaical' Greece , — th^ -right knowledge leads to 
right action. Only in recent decades have soc^^al scientists ' at- 
tempted to evaluate the complex .process *of personaliT&v^^and skill 
development of late adolescence. . ^, 

As in most educational settings, the establishment 'of an 
experimental design ta' control^ most 0/ the variables, which migfit 
af-fect desired outcomes ^^ooms as -an insurmountable task. The de- 
signers of programs funded tHrough the Lilly Endowment did not^ ^ 
concern themselves with experimental design per se, but primarily 
focused their efforts on program desigfii a.nd implementation. 

A research desi,gn that, would adequately ^cc6unt for vari- ' 

i • , ' . - • ' , 

ables would require longitudinal studies, requiritw a period of 
time beycJnd the pei^iod of -funding. Their concerns ldy>^marily 
with formative evajLuation which we interpreted as a quick as; 

*& V 

ment of strengths and weaknesses .for the purpose of improving the 
program. \ • . ' ' ^ 

In the majority of instances, program directors could not 

^^establish a control gr^up by which to compare the experimental 

. ■ ^ ^ 

gitoup.>^Stuaents who participated in' the programs came from a con- 

tinuum with extreme positions. In many instances they elected 

through interest' in the specific educational experience And the ■ 



anticipated and direct benefits they see accruing td themselves ; 
at the otber- end -they had no choics^but to^ fulfil a- college 're- 
quirement presc^ribed for all student^x^ Their mentors did not 
env:^siiG>n an experimentally controlled atmosphere in which the 
expferienc^s^ould take place. . » ■ ' 

V. Another major 6l?stacle derives, from faculty attitudes with • 

" ' ' ' . • , r ' , - 

-r^egard to evaluation." Most persons 6onnecte.d wijth the programs 
were not trained in research methodolc^y nor did they envision 
themselves^ becoming proficient iij that area. Many expressed doubts 
al?out th« effic^ijy of ">ni^ evaluation methods, and a few rejected such 
a proc^ c>li the" phllo^phical .grounds that true education defies 
measurem^^H^ To jbrove to others what they themseivesx-judged 
successful froiKtheir own' experience and intuition seemed to them 

an unessential -burden. ) 

** if 

As program directors, .they expressed concern with the cost 
and th^ energy of implementirig, an evaluation program, when they 
redlly ^desired us,ing their). resources for program development. 
Their prior experience.with outside evaluators had, for the most 
part, been non-productive. They did not see -hoW evaluation could - 
lead tA the improvement of ^ their program and looked upon j.t 
primarily as meeting the accountability rdquiren^ents , necessary but 
painful, of a granting agency. They did hot viei; evaluation as 
formative which could lead to program imp'rovement. and a i)etter 
use of ftnds granted to them. 



. The model which we proposed fco '^them an'^l<;^pated most of the 

objection^ cl^fe<3^bove and incorporated them into the evaluatipn ^ 

procedures. &asiC:ally, our^ desired outcomes were: (l)^:ollection 

x>f data onVhich to around ,decisioa-makinq ,with reasonable e>bendi-' 

ture of resources; (2) Documentation of evidence so that outside> 

y^dqw5 oe^sitS make quasi-independent judgments and be^ persuaded b^ 

•* ^ " 

the statements of insiders; and, (3) Cooperation with program staff 

with regard to the development of evaluation design, self-ex/aluatipn*, 
and report writing. ^ , . • 

Tor this , we made an adaptation of a perception based model 
of evaluation grounded primarily on the work- of /Richard Kunkel and 
others at Saint Louis University and the CIPP model of Daniel 
Stufflebeam, which speaks to the analysis of a program with regard 
to context, input, process and product. We arvticipat^d that th-fs 
approach would b^ mucPl -fess threateiiing to^faculty membert, would 
involve tihem in: the entire process^ and. would indicate to them ways 
in which they could jlmprove their b^sic program during its begin- 
ning stages. . r 

Serving 'as a touchstone of pur perception baaed model was 
the belijef that' in most inst^^nces the objectives, as defined by 
the project directors, can not be measured by standard instruments. 
Our perception 'based mod^l maintains that consensus about the value 
and quality components of 'a program represents an approximation to ; 
an objective measurement of success that cannot "be discounted when 
the persons ihvolved in the educational experience agree that. they 
are achieving ^e^sired benefits. ' . , 




To "-meet, the initial objections of many, project directors 
th regard to evaluation, we set as our goals the following 
characteristics: "First, that the prdcess be. helpful an4 that the 
personnel involved in the programs recognize tha^fe our procedures 
would benefit them by stimulating them tb think in terms' of" im- 
proving student learning. Our emphasis lay in formative evaluation 
with constant revision of the program in light of new information 
with, regard to its effectiveness ss an ongoing ^process • We* did^not 
V stres^procedures or acceptance of ^ata unless the program admini 
. ttators would accept it arid use it' in deci^j-on-making both for th 
immediate and long term duration of the program. * 

Th^t the proce^s^ be reasonable in. terms of expending human 
energy, time, and financial ^expenditures. Our "hope was to ptovide 

• maximum .information about. the ef,£ec^s of projects with miriimuiti 
inputs ,This"met the concern "o^ pro j-fect directors that nearly ali 
of the money granted -to them'^e used .in planning activities to ' ' 
bring about change, in students . and , that a reasonably -minimal amount 
would go to monitoring the process . Thus, we did- not ,enc6urage ' 
over-u$e of standardized, instruments- which ^re costly to administei^ 
a^d difficult td analyze in terms of the program's direct eifect on^ 
students . - , . ' ' ' 

Third., that our approach wbul4 be holistic that' is, a 
great variety of "sources of evidence with regard' to success of the, 
program.. would be investigated 'and admitted into th^ evaluation pro- 

' ' • \ ■■ ■ , . ; 

• pednre. This would indlude' percW)tions of students' from a variety 




of sources, fibservation of t^e" process ^observable conunitment given 

by persons in'^support of the ^ogram, comments , ^ letters , and 
^critical incidences— all clear ]\ indicating tV^the program directors 

that the ajg^ivities were prized by ri 

chanjge was Observed and recognized by 

question raised her6 was, what -kind 

as markihg success in your' program,* 

usual it might "Bj^v^J^^ . ' ■ ^ 

( Fourth, that the inter-persoria^l 

negotiation at each step of thfe^way. Instead of intruding oiir 

objectives, values, and" measiinements of ^outcomes into their system, 
we discussed these concepts with the s,taff ascertaining ^wh^t mah,i-> 

ffestations of these "they fell^^ essential for the' 'program. If there-N 
was a discrepancy between their views 'and ^urs, which we as con- 
sultants could not* accept; these ^were- negotiated to everyone's j 
mutual satisfaction . ^ That process, also applied to , the. evidences 
that would be -acceptable as marking progress a^ well as\o final 
reports submitted* to the Endctwment. Tn this latter case, when 
Agreement on specific points; could not be reached, the institution 
re-sponded to our . critique of their self-evaluation with, one of 
their h\ffn. Our approach was .^committed to dialogue"' with regard to 
all aspects of evaluation, procedures never stressing our own 
biases, but always pushing them to clarify''" and explain why the • . 

" ' • ; .- " - ' 

objectives and' evidences which they desired were, . indeed , valuable 

' . '-'.',>- ■ - 

and valid 'with regard to .pr(;igrant goals.- 



1 Fifth/ that the anafyses -of information be moved in a ' 
direction of greater quantification, wfien. the Sases of data woyc ' 
attitudinai;^^nd, subjective as ^^ndicated hy self-evaluation 
techniques. Essentially we desired thafe the evid^nces^ used in. 
judging the program be documented in »some ,way ito be suraxnar ize^d 
and ass i^nilated more easily by a non-involved third' party. So 
often when we began to talk about anthropological evoidisnces of 
success / project directors indicated that they could recall- many 
items from thdir past experience which would indicate achievement 
of stated' objectives . However, they could Qot verify thes^e. by 
producing documentation that woul^^convince us of the p,ersuasiye- ' 
ness of ithese attitude^/ ** . 

e . The actual process of evaluation proceeds accordingly: 
As e\^aluator consultants' wfe visited .and arranged\to negotiate ^all 
aspects of the ' prbcess ^with personnel invblved in the programs, 
We'^raised questions as' follows: W^at major qug's£ions should be 
asked i,n the evaluation ctnd what arc .the objectives' of * the ex- 
perience?^ ^^Ji^at sources of d'vidence can he fo'unji with regard^ to the 
prercess?* What quality components ban we apply to 'the evidence * - 
received thr<>ugh 1;he various processes of Qata gathering? Who ' ' . . 
aire *the best judges a*f the process and how much weight ought one- ' 
to give to the evidence derived from themJ 

We reques"tedt materials about the progranv^and evaluative' ' 
data and reports; We strove through letters, tel^eghohe . palls , ^ 
and the on-site visitatioh to build mutual understanding and trust 



for -our roie in rbl|^ evaluation process; We gave, feedback after 

^our f ifst ..meeting arf^ t^-g receipt of, their self-evaluation reports 

»" '"v -■ . . * - * ' ' ' 

and encour'g^ed tiiem to ^e. candid in appraising our ^ ef forts> and 



■ \ 

activities, ' 



\^ The programs from three of the Continuing Conference school 
provic3^^ examples by which to demonstrate our. approach. In judging 
the success qf internships, we developed several means grouniied 

on perception and affirmed by project personnel as valid appraiser 

' " ' ' ' ^ f ' 

of t:h^ process. Through the' use of a log writtenjby students on a 

daily or weekly basi^^e hoped to encourage students to reflect 

in written form about tk^eir experiences, emphasizing -jiot ^ ;inuch 

their performance of activities but focusing on their attitudes 

toward the job environment;., values issues whi^h emerged, their 

emotional tfeelings about .them, an^a insights that occurred with 

regard to .decision-making while. engaged in their" work situation. 

An ana4.ysis of tlie Mog^ over a period of weeks, or months would 

indicate the depth of thinking and insightf ulness which a student 

was undergoing. /The tiritjeria by which to m^a^ure progress were 

distilled from' the overall objectives of ' the' internship component 

of the pi^ogram. ^ We encourage^; the evAluators to describe' a pro- 

gression of step^ thAoUgh which the student would pass as a way to 

assess individual as \^11 a^ group development'^ 

We- encouragedlj^ development of "benchmarks'^ as a means . 

of recordin^^, at^itudfes towards the process of the internship by 



the instructor, student, and the supervisor • These are? brief 

■ ■ . ' ' - . . ■ ■ • \ 

instruments utilizing Likert scales and' open-ended question^ to , 
elicit 'judgments oX^^ satisfaction and self-evaluatipn of progress _ 
toward individual obje(^tiVes, designed for use at frequent inter- 
vals so 'that^ the results >qan be used to alt^r the activities or 
r;elationships throughout the length of an experiejice. * ^ 

J We felt that an interview with both supervisor 'and student 

would yield deeper "^insights to behavioral and attitudinal changes- 
and permi^t comparisons of twq different perceptiojis of th§ same 
events. We suggested a directed interview, the re'sults of which 
could be documented and tabulated in quantifiable manner.. 

The simulation program implemented at Denison University 
lends itself easily to. the use of *^benchmarks" . Both students and- 



instructors could use such jcLhstmaii^ents as a qui^ wayvof register- 
irig their satisfaction with the unique teaching methods used. 

* . ^ . , - • ... ^ . ^ \ 

Observati,on^^of the Simulation in a classroom presentation o^^rs 
another source of evidence for consideration'. The observer .mi^st 
construct fcriteria by. which he judges the quality of the experience 
for students in consort with^the instructor based on the objectives 
of the particular simulatidn involved. . Self nervaluation in greater 
depth fi:om students can' be learned from requi'ted essays or^ inter- 
views during or at the conclusion of . the*/e:KPerience . Again^ 
criteria ought '*to iDe- , set after cAref ul- considerati^on* of objectives/ 
Some. way to document the findings otight to be employed so inforwa- • 
tion gainedv can be offered to an - outsider in a per suasive. format . 
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^ Not surprisingly, the most difficult programs to, evaluate 
are those that, aspire to educate the liberated individual. ^ So 
often in cases of enculturation programs, one is askdd to evaluate 
,a philosophy^ of education rather than an educatiQnal process. A 
traditional^ view' of higher e^cation has jnaintained that exposure 
to the great idea^ of Western thought is in itself an elevating 
experience • ' The goals of such curricula are defined in conceptual 
tefins that cannot be judge^ in any given tijne *frame. 

While granting the evanescent' nature of the outcomes of 
suci^ programs , we argue that some approxi-matiqns of the success, of- 
* tM approaches tak^n can be learned through use of a perception 
ba'sed model. , found some reluctance to undertake an instrumented 
approach even though the testing and measurements are^ has many of 
these ^o offer. Basic reasons for rejecting this tack follow from 
, financial and energy constraints, but , perhaps more importantly 
from a distrust that prepared instruments really measure the 
qualities desired for the students 6f specific course. I^sti- 
'tutional philosophies and the special desires of the faculty 
' involved seem not to admit the efficacy of a Vcanned" treatment. 

Adaptations of methods previously described, that is 

<• » 

, "benchmarks", interviews, logs, assays, and' observation ^tre all 
flexible enough to meet the needs of evalu^ion. I'he crucial steps 
here are defining- the/ criteria by which to measure progress. TKe 
perception based model asks the question what would you accept as 
evidence of success, and who is the best judge of the process. 
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So with the other two programs we pressed individuals to state 
their objectives in bel^avioral activities that could be observed 
or in attibudinal terms that could be self-evaluated and recorded. 

To judge progress made by st^adents, the faculty were asked 
to establish for themselves a developmental schema consistent with 
their objectives. As models we suggested the works of Lawre^ce 
Kohlberg, Abraham Maslow,'Erik Erikson, William G. Perry, and the 
seven vectors of Arthur Chickering. Out of this background a 
paradigm compatible with the specific program could emerge, pjfo- 
viding the criteria for personal, growth along continua important 
to the designer of the curricul'iiin. . i 

^ The results pf bur approach with the .Continuing Conference 
Institutions to-da^te have been mo s,t5" rewarding and, Weifeel, bene- 
• ficial to the institutions' involved. Becaulfe^, of bur approaclf*T:r>-- 
various methods of evaluation which involves tha^ particip^irits in 
the process of defining goals and objectives^ we feel that .trust 
has been established both for us and the process / and that the' 
outcomes of evaluation are accepted. One aspect of our' approach 
mentioned by more pro"3ect personnel than any other as of great 
service to them was our insi^tance upon clarification of objectives 
and re-defdning goals into behavioral and attitudinal terms which 
could be observed, measured through instruments, or self-evaluated 
in spme meaningful way. ' 



(13) 



Wh regard to" the process itself, we feql.that ^ working 
session with faculty, taking them through the steps of evaluation,. 
yie:ids more benefit than our 'observation or a walk-thru of their " 
^ particular project.' " We stressed^ that .our opinion about the pro- 
^gtam was of little importance, of greater cohcern Las their ability 
•to-set objectives, define criteria, and establish /acceptable ' 
evidences. • With Regard to^ the. .final report of th4 evaluation, we 
learned that the most h^-lpful format occurred whJn'the personnel 
:invoived wrote -their own' self-evaluation after oir .conferring with 
^ them,- to which we -responded in the form of a gritique'of the 
evaJwation 'procedures rather than, to the progran itself . .^Oa • . 
those f^w occasions wh^re we 'wrote the report f rm /^he- information • 



groups expressed 



given to us, by the individual, institutions, bb'tii 
dissatisfaction. Theyj argu^d^a^t we' did not' f lilly understand^ 
their activities and^oals; we fSlT^^^es^ed bec4use it required 
more-fie^iations to/airrive at a. point of "ifrti^uia^J satis faction .- 
The. institutional sel^-evkluation rfeport encourages, the pragxam 
staff t9.enga^ge in the development ,of their' ^wn evalua'tion pro- - 
cedures from the beginning and blocks reliance on "^ou^tsiders to 
come in and state what is right or wrong with the -program. The 
use of outside "objective" evaluators often creates a defensive ' 
attitude in which a -great amount of time. and enfergy is' expended, i^' 
defending the' negative aspects of -the program, or. explaining it . 
away by one mtf&ns or another^ . ' • " ' " ' ; ■ , 
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